Last updated on 2017-03-17
arrange() & _joinfilter() rows/observations matching criteriasummarize() numerical variablesgroup_by() group rows/observations by a categorical variablemutate() existing variables to create new onesarrange() rowsAnd _join!
Really simple. Either
DATASET_NAME %>% arrange(VARIABLE_NAME) orDATASET_NAME %>% arrange(desc(VARIABLE_NAME))library(dplyr)
# Create data frame with two variables
test_data <- data_frame(
name=c("Abbi", "Abbi ", "Ilana", "Ilana", "Ilana"),
value_1=c(0, 1, 0, 1, 0),
value_2=c(4, 6, 3, 2, 5)
)
# See contents in console
test_data
Run this code. Notice the subtle diff between 2 and 3:
# 1: Arrange in ascending order test_data %>% arrange(value_1) # 2: Arrange in descending order test_data %>% arrange(desc(value_1)) # 3: Arrange in decending order of value_1, and then within # value_1, arrange in ascending order of value_2 test_data %>% arrange(desc(value_1), value_2)
And now the last component of data wrangling: joining/merging two data sets. Run the following:
x <- data_frame(x1=c("A","B","C"), x2=c(1,2,3))
y <- data_frame(x1=c("A","B","D"), x3=c(TRUE,FALSE,TRUE))
x
y
We join by the "x1" variable. Note how it is in quotation marks.
left_join(x, y, by = "x1") full_join(x, y, by = "x1")
join (right-hand column of back of cheatsheet). To keep things simple, we'll try to only use:
left_joinfull_joingroup_by() & 5MV#4 mutate()filter() rows/observations matching criteriasummarize() numerical variablesgroup_by() group rows/observations by a categorical variablemutate() existing variables to create new onesarrange() rowsRun the following in your console:
library(dplyr)
# Create data frame with two variables
test_data <- data_frame(
name=c("Albert", "Albert", "Albert", "Yolanda", "Yolanda"),
value=c(2, 2, 2, 3, 3)
)
# See contents in console
test_data
group_by(name) puts grouping meta-dataRun the following. Notice the data itself doesn't change, but the data about the data does:
test_data test_data %>% group_by(name)
Run both these
test_data %>% summarise(overall_avg = mean(value)) test_data %>% group_by(name) %>% summarise(name_avg = mean(value))
What's the difference?
Chalk talk
Here:
Mutate existing variables to create new ones. Always of the form:
DATASET_NAME %>% mutate(NEW_VARIABLE_NAME = OLD_VARIABLE_NAMES)
Using the same example as earlier. Run both:
test_data %>% mutate(double_value = value * 2) test_data %>% mutate(double_value = value * 2) %>% mutate(triple_value = value + double_value)
%>%, 5MV#1 filtering, and 5MV#2 summarize()%>%Piping allows you to
filter() rows/observations matching criteriasummarize() numerical variablesgroup_by() group rows/observations by a categorical variablemutate() existing variables to create new onesarrange() rowsfilter() rows/observations matching criteria
Take flights and then filter for all rows where year is equal to 2014.
Note we use == and not =
library(dplyr) library(nycflights13) data(flights) flights %>% filter(year == 2014)
summarize() numerical variables using a many to one function:
Examples of many to one functions:
sum(): sum of n valuesmean(): mean of n valuessd(): standard deviation of n valuesWhat's going here?
library(dplyr) library(nycflights13) data(weather) weather %>% summarize(mean_temp = mean(temp))
With the internet, we are in a new age of data:
Jenny Bryan said: "Classroom data are like teddy bears and real data are like a grizzly bear with salmon blood dripping out its mouth."
| Traditional Classroom Data | Real Data |
|---|---|
Some attributes of real data:
Inconsistent formatting is a real pain:
To take this, we now officially introduce the dplyr package: a grammar of data manipulation
function() you use.Say hello to the 5MV: the five main verbs
filter() rows/observations matching criteriasummarize() numerical variablesgroup_by() group rows/observations by a categorical variablemutate() existing variables to create new onesarrange() rowsAlso, later _join() two separate data frames by corresponding variables
Recall from first Grammar of Graphics lecture, we displayed
Say these piecharts represent polls for a local election with 5 candidates at time points A, B, and C:
Answer the following questions:
geom_bar() is the trickiest of the 5NG, so we'll use it in limited capacity.Two different ways to have counts show on y-axis:
geom_bar()data in a variable count, n, etc.Counts are not pre-computed:
| Row Number | name |
|---|---|
| 1 | Albert |
| 2 | Albert |
| 3 | Albert |
| 4 | Mo |
| 5 | Mo |
Counts are pre-computed in variable n. So n becomes a y aesthetic variable!
| name | n |
|---|---|
| Albert | 3 |
| Mo | 2 |
ggplot2?If I know your name, I can guess your age. Looking at the handout answer the following questions:
As of Jan 1st, 2014 in the United States
Chalk Talk: Age of 544 Members of 113th United States Congress:
From okcupiddata package, the profiles data set:
Restricted to heights between 55 (5'5'') and 80 (6'8'') inches:
x aestheticy aestheticFor values: \(-2.5, -1.5, -0.5, 0.5, 1.5, 2.5\)
Let's draw histograms using the following binning structures:
Facets allow you split ANY plot by a categorical variable. In this case by adding +facet_wrap(~sex) to the ggplot() call
| A | B | C | D |
|---|---|---|---|
| 1 | 1 | 3 | Hot |
| 2 | 2 | 2 | Hot |
| 3 | 3 | 1 | Cold |
| 4 | 4 | 2 | Cold |
A statistical graphic is a mapping of data variables to aes()thetic attributes of geom_etric objects.
ggplot(data=simple_ex, aes(x=A, y=B, size=C, color=D )) + geom_line()
What's not great about this plot, especially near (0, 0)?
This is called overplotting: when points are stacked so densely we can't see what's going on!
There are two ways of dealing with this:
A statistical graphic is a mapping of data variables to aes()thetic attributes of geom_etric objects.
The five named graphs we'll see in this class. Note: I reordered them from last time to be easiest to hardest to work with:
ggplot2 packageIn tidy format:
| A | B | C | D |
|---|---|---|---|
| 1 | 1 | 3 | Hot |
| 2 | 2 | 2 | Hot |
| 3 | 3 | 1 | Cold |
| 4 | 4 | 2 | Cold |
In 1812, Napoleon led a French invasion of Russia, marching on Moscow.
It was one of the biggest military disasters ever, in particular b/c of the Russian winter.
Famous graphical illustration of Napolean's march to/from Moscow
This was considered a revolution in statistical graphics because between
there are 6 dimensions of information (i.e. variables) being displayed on a 2D page.
A statistical graphic is a mapping of data variables to aes()thetic attributes of geom_etric objects.
| Where? | data |
aes() |
geom_ |
|---|---|---|---|
| top map | longitude | x |
point |
| " | latitude | y |
point |
| " | army size | size |
path |
| " | army direction (forward vs retreat) | color |
path |
| bottom graph | date | x |
line & text |
| " | temperature | y |
line & text |
| 2005 - Proposal | 2009 - R Implementtation |
|---|---|
From ggplot2movies package, the movies data set:
From nycflights13 package, the flights data set:
From okcupiddata package, the profiles data set:
From fueleconomy package, the vehicles data set:
From babynames package, the babynames data set:
Say hello to the 5NG: the five named graphs

The nycflights13 package contains "tidy data" all 336,776 flights that departed from NYC (e.g. EWR, JFK and LGA) in 2013.
To help understand what causes delays, it also includes a number of other useful datasets.
weather: hourly meterological data for each airportplanes: construction information about each planeairports: airport names and locationsairlines: translation between two letter carrier codes and namesIn small teams, take 3 minutes to write down
Recall the tradeoff:
| Less of this… | More of this… |
|---|---|
You need to install each package once.
You need to load a package everytime you want to use it.
library(PACKAGENAME) in the console.Today's Learning Check: Install and then load 3 packages:
dplyr: a package for data manipulationggplot2: a package for data visualizationbabynames: a package of baby name datababynames PackageThe babynames package contains for each year from 1880 to 2013, the number of children born of each sex given each name in the United States. Only names with more than 5 occurrences are considered.
Have students engage in the data/science research pipeline in as faithful a manner as possible while maintaining a level suitable for novices.
We will, as best we can, perform all this:
And not just this, as in many previous intro stats courses:
Foster a conceptual understanding of statistical topics and methods using simulation/resampling and real data whenever possible, rather than mathematical formulae.
In this course, computers and not math will be the "engine". What does this mean?
Blur the traditional lecture/lab dichotomy of introductory statistics courses by incorporating more computational and algorithmic thinking into the syllabus.
go/rstudio/ (on campus or via VPN)Develop statistical literacy by, among other ways, tying in the curriculum to current events, demonstrating the importance statistics plays in society.
Either
| R | RStudio | DataCamp |
|---|---|---|
go/rstudio/ with your Midd accountNow we will use R via DataCamp instead of via RStudio, but just for driver's ed. Two panels exist in both: